A fixing device conveys a paper with nipping the paper between a heat roller and a press roller, and peels off the paper, which is subjected to a fixing processing and is wound around the heat roller, from the heat roller by an arm-shaped peeling member with a peeling claw arranged at the downstream side in the conveyance direction (for example, see Japanese Unexamined Patent Application Publication No. 2012-98511). The peeling member is swung freely by a pivot bearing at the center part thereof, and the peeling claw is arranged on one end of the pivot bearing, the other end of the pivot bearing is energized by a spring and the like. In this way, the heat roller rotates in a state where the peeling claw is slidably in contact with the roller surface (outer peripheral surface of roller) of the heat roller. The paper surface on which the toner is carried is in contact with the heat roller to be heated and pressed, and thus the toner is melted and the toner permeates the paper. The toner has viscosity, and the toner that cannot be fixed on the paper adheres to the roller surface of the heat roller. When the toner on the roller surface is scraped by the peeling claw, the toner sticks to the front end of the peeling claw slidably in contact with the roller surface. Conventionally, there is known an image forming apparatus in which the rotation of a fixing roller is stopped temporarily according to the integration value of the printing ratio, and the toner adhered to the peeling claw is transmitted to the fixing roller (for example, see Japanese Unexamined Patent Application Publication No. 2009-92683). In an image forming apparatus capable of carrying out duplex printing, through the speed control to the heat roller and the paper discharge roller pair, the rubbing of paper by the peeling claw when the paper subjected to simplex printing is switched back and conveyed is not generated (for example, see Japanese Unexamined Patent Application Publication No. 2006-251178).